feat(shadow-requests): add experimental shadow_config wrapper - #250

_Evaluated forwards ConfigInstance's constructor fields by hand. If ConfigInstance gains an option later, the served run under shadow_config() would silently run without it, and its output would stop matching config(). The parity tests wouldn't catch that, since they only exercise today's options.

A small test comparing the parameters of ConfigInstance.__init__ with the fields forwarded here would make that drift fail loudly.

Updated 🙂

This repeats the client half of extract_variation (init_client, get_client, to_ld_context and the awaitable handling), so a later change there, such as to lazy init or context conversion, won't reach shadow_config().

Could core have a private helper that fetches the raw value, with a generic with_reason flag, called by both extract_variation and this function? It doesn't name the feature, so core stays unaware of shadows.

Updated 🙂

Minor: when streaming, shadows start here, before judges run (_stream_events runs them after this generator finishes). On invoke they start after _invoke returns, which is after run_judges has finished. Neither delays the caller, but with slow judges, invoke shadows start noticeably later than stream ones. Fine if intended; a short comment saying so would help.

Updated 🙂

On failure, ShadowResult.track_data is task.track_data, which has only the six shadow fields (runId, isShadow, primaryVariationKey, shadowRequestId, ownerKind, ownerId). There is no configKey, variationKey, version, or model and provider names. That applies here, on the timeout path above, and when no handler matches (select_handler raises inside the try). On success it's B's full tracking data. So a worker running deferred tasks can't tell from the result which variation failed.

The JS SDK returns B's full tracking data on every path (makeRunTrackData plus the shadow fields). Here it would need the same payload execute_and_track builds inline. A small private helper in tracking.py, used by both, would keep them from drifting. The tests at lines 749 and 766 only check isShadow, so asserting configKey and variationKey on a failed result would cover it.

Minor, if you want the SDKs aligned: the no-handler error reads "Handler for provider X not found" here and "No handler for provider X" in JS.

Please check deadline.expired() after the async with block too. asyncio.timeout only raises TimeoutError if the CancelledError propagates. A handler (or a provider client) that catches the cancellation for cleanup and then returns a value exits the block normally, and its late result is tracked as generation:success. A result that arrives after the timeout should be discarded and tracked as a timeout:

async with deadline:
    result = await handler(*args)
if deadline.expired():
    raise _ShadowTimeout("timeout")
return result

I confirmed the asyncio.timeout behaviour on its own, but didn't run it end to end through run_shadow. A test with a handler that swallows CancelledError would pin it down.

When the exception has no message (a bare TimeoutError() is common from HTTP clients), str(err) is "", so a worker that checks if result.error treats the failure as a success. Suggest str(err) or type(err).__name__.

Optional: this test and the timed-out test below only check that generation:success appears among the served call's events. Comparing them to the events from the same call with no shadow configured would prove the shadow left them unchanged.